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Documents are available from the Educational Policy Research Center at Syracuse 
in three formats, besides the regular publication. Notes on the Future 
of Education: 



RESEARCH REPORT S 

Reports which have completed review by the EPRC and which deal with 
specific, policy oriented research. The reports in this series are 
usually marked by intensive research, either quantified or historical, 
and address themselves to specific research questions. 



EXPLORATORY REPORTS 

Reports which, while dealing with policy issues, often approach the 
realm of conjecture; they address themselves to social issues and the 
future, may be prescriptive rather than descriptive in tone, and are, 
by nature, more controversial in their conclusions. The review of 
these reports by the EPRC is as rigorous as that for Research Reports, 
though the conclusions remain those of the researcher rather than 
necessarily representing consensus agreement among the entire Center 
staff . 



WORKING DRAFTS 

Working Drafts are papers in progress, and are occasionally made 
available, in limited supply, to portions of the public to allow 
critical feedback and review. They have gone through little or no 
organized review at the Center, and their substance could reflect 
either of the above two categories of reports. 



The research for this paper was conducted pursuant to Contract No. 
OEC-1-7 -07996-4253 with the Office of Education, U.S. Department of Health, 
Education, and Welfare. Contractors undertaking such projects under Govern- 
ment sponsorship are encouraged to express freely their professional judgment 
in the conduct of the project. Points of view or opinions stated do not, 
therefore, necessarily represent official Office of Education position or 
policy. 
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ABSTRACT 



I. In recent years the development of rational methods for studying 
the future has accelerated dramatically. The cross-impact matrix 
method (CIM) is one of many such methods, and in many ways the problems 
of CIM are typical problems for futures methodology. In this paper 
I take a close look at CIM, and what I see as the basic assumptions 
behind the method, to determine what role it should play in helping 
men to think about the future. I try to show why I think it should 
be used with the utmost caution as a decision-making tool. At the 
same time I argue that this does not preclude its potential value as 
an heuristic device. 



II. I outline what I see as the major problems with the CIM computer program 
as developed by the Institute for the Future and used at the Educational 
Policy Research Center: 



A. Problems having to do with the formula: 



P 



B 



p r + ap b 



P B + (1 * V S AB 




1. The time factor need not, and in some cases should not, be 
linear or decreasing. 



2 . 



If the initial probability 
the change in probability AP 

B 



of event B is less than 0.5, 

is the opposite of what it should be. 



3. If the year of 50% probability (yfp = t ) for event A is not 

between year 0 and year 2t (where t = time horizon) , then the 

final probability P ’ may be greater than 1 or less than 0. 

B 



4. The quadratic nature of the formula places undue constraints 

on possible change in probability AP . 

B 





5. The yfp of event B ought to be a factor in calculation of the 
change in probability AP p . 

B. Problems having to do with the order in which events are assumed 

to occur. 

1. Potential order of occurrence (ooo) of events should be taken 
into account, and it should be recognized that not all ooo ' s 
are equally probable. 

2. I analyze how the present program handles ooo, and try to infer 
from this the basic assumptions upon which the method is based. 

3. The monte-carlo nature of the program necessitates a large 
number of "games." I suggest some ways of systematically 
cutting this down by as much as a factor of 100. 

4. I suggest some alternative ways of generating "better" ooo's. 

C. Problems of specifying the input and how that input fits into the 

formula: 

1. Some necessary assumptions require us to reconsider our definition 
of "impact" S^g. 

2. Some alternative definitions are suggested. 

3. We also have to address the question of the meaning of 
"probability" as we treat it. 

III. Having examined some of the epistemological problems with CIM, which 
must influence the way \,a employ the method, I briefly discuss the 
use of CIM as an heuristic device. I suggest a simple alternative 
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procedure which is easier to understand and is less likely to produce 
results which might be misconstrued. 

IV. The final chapter of the paper is an attempt to look at the contents 
and nature of the foregoing in the light of the whole area of futures 
methodology. I suggest that many problems with CIM arise generally 
among rational methods for studying the future. As examples of the 
kind of problems we should address, I discuss (1) simulation, (2) probability, 
and (3) judgments based on knowledge which is not made explicit. I 
suggest that the future of futures research might best lie in research 
on problem areas such as these, rather than in the development of new 
techniques which gloss over them, or ignore them completely; we might 
find that much of so-called "scientific" futures methodology is based 
upon scientifically indefensible foundations. 
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INTRODUCTION 



"In the days of Kronos and when Zeus was newly king,"* he attempted to 
destroy mankind by giving us absolute knowledge of the future, for he knew 
that it would break man's spirit to know the circumstances of his death. But 
that black sheep of the Titans, Prometheus, had taken a liking to us mortals, 
and ever since we have had problems with seeing into the future: 

Prometheus 

. . .1 rescued men from shattering destruction that would have 

carried them into Hades' house. . . 

Chorus 

Did you perhaps go further than you have told us? 

Prometheus 

I caused mortals to cease foreseeing doom. 

Chorus 

What cure did you provide them with against that sickness? 

Prometheus 

I placed in them blind hopes. 

Chorus 

That was a great gift you gave to men. 

Prometheus 

Besides that I gave them fire. 

Chorus 

And do creatures cf a day now possess bright-faced fire? 

Prometheus 

Yes, and from it they shall learn many crafts.* 

These and other gifts of Prometheus transformed man to a reasoning 
being. Alas, whether we like it or not we had no choice those many years 
ago but to trade off our prophetic powers for survival and craftiness. Lacking 



Aeschylus, Prometheus Bound (Translated by P. Vellacott. Penguin Books, 
Baltimore, 1961). 
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the power of prophecy, and being possessors of rationality, we have over the 
years invented many devices for predicting the future' and in the last two decades, 
as faith in our ability to use the promethean gift of rationality has increased, 
the development of scientific methods for studying the future has accelerated 
dramatically. 

A. Rational Forecasting Methods 

A recent survey* cites no less than twenty-four such methods, including 
everything from econometric modelling techniques to game playing, currently 
being employed for generating every imaginable kind of information about the 
future. I suspect that a systematic study of these many different methods would 
identify a number of recurring concepts, among them: probability (personal 

and objective), plausibility, exogenous and endogenous variables (in a social/ 
technological environment), the extension of past and present trends, interactions 
among future events, time of and order of occurrence of future events, and tacit 
(untellable) knowledge. 

The question that has to be asked about these concepts, and it has to 
be asked every time somebody invents a new method which involves them, is how 
capable are we of dealing with them rigorously? Do we, for example, have a keen 
enough understanding of and agreement on the concept of personal probability 
to make it a working parameter in a precise research method? Or do we regard 
personal probability as extremely variable and the method therefore as 
extremely imprecise? 

B . Motivation for the Paper 

The cross-impact matrix method is one which uses many of these concepts; 
it is typically rational and typically systematic, and typically it relies 
heavily on a precise command of the concepts. The cross-impact matrix method 
(CIM) was developed by T. J. Gordon and 0. Helmer about four years ago as a tool 
for describing and analysing interactic s among a set of possible future events. 

& 

A 

John McHale, "A Survey of Futures Research in the U.S.," 

IV, 5 (December 1970), p. 201. 




2 8 



The Futurist, 



Most of the original work on the development of CIM was done by the 
Institute for the Future, and I recognize the valuable research still being 
conducted there in developing and refining the method.* 

I am concerned , however , about what I see as a developing over- 
optimism about the usefulness of the method as an analytic tool for decision- 
making. In our haste to find quick solutions to the problems of forecasting, 
we may be misinterpreting the cross-impact output as hard and reliable data, 
when in fact there may be no basis for doing so. We should first ask just 
what the CIM can do, what its shortcomings are, and what contributions it 
can make to futures methodology. 

In this paper I can only ask a few of the many questions that should 
be asked of CIM, and these will deal mainly with problems that have occurred 
to us in the two years we have been experimenting with the method at the 
Educational Policy Research Center (EPRC) at Syracuse. I feel that this kind 
of critique represents the sort of analysis we should make of many new futures 
research techniques before we decide how or whether they may be useful. 

I hope that the paper does not have too negative a tone, for there is at 
least heuristic** value in playing with quantifiable models for studying 
the future. By developing and working with new technioues, as well as by 
criticizing those techniques which already have been developed, a contribution 
may be Inade toward the advancement of valid futures research methodologies. 

I do object to the lack of stress placed by its developers on CIM's 
inconsistencies and flaws, and this is the motivation for this paper » To 
present CIM without emphasizing its drawbacks is in the long run to play on 
modern man's faith in the omniscience of the black box. The formulae upon 



See Appendix A for a summary of work done at IFF to April, 1970, reported in 
the Use of Cross-Impact Matrices for Forecasting and Planning , by Rochberg , 
et. al. 



By "heuristic" in this paper we mean "stimulating greater insight into a 
situation. " 
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which the model is based are not so complex that the layman cannot be made to 
understand them; it is therefore essential, since the model is so imperfect, 
that the formulae be explained to the user. This is especially important if 
there is a possibility that the user will, in his turn, educate others in this 
supposedly useful methodology. Perhaps this paper will clear the air so that 
we can begin to make some positive contributions to futures methodologies 
through the use of the CIM. 

C. Focus of the Paper 

In the time that the cross-impact matrix (CIM) has been at EPRC many 

questions have arisen about the usefulness of the CIM methodology, both as an 

heuristic tool to be used in exercises designed to enhance one's ability 

to think about the future and as an analytic tool to be used by planners to help 

in actually forecasting future occurrences. This paper makes no judgment on the 

use of CIM as an heuristic tool; that the method of cross-impacting can provide 

a stimulating and possibly positive learning experience has been shown many 

times in exercises conducted by Larry Hudson and Stuart Sandow of the EPRC 

and by the original developers of the method from the IFF. In this sense, CIM 

has been used successfully to "improve communications and understanding of the 

11 # 

complexities involved in the planning and decision-making process. 

However, in my opinion, CIM has not yet been used successfully as an 
analytic technique. In fact, I feel that the problems with CIM are so basic 
that it will not be usable as an analytic tool until they are solved. The 
purpose of this paper is to explain some of these problems and their importance 
to the successful application of CIM. 

(Note: It is not inconsistent that a method might be useful heuristically 

and yet be invalid analytically: it is one thing to stimulate people to sharpen 

and expand their thinking about the future by introducing a responsive, oracular 
black box — however valid it may be — against which they must defend their percep- 
tions, but it is quite another to build a black box which not only is responsive 
and oracular, but also provides a consistent and correct analysis of possible 



*S. Enzer, A Case Study Using Forecasting as a Decision-making Aid, p. 3. 



futures. When the method is to be used for purely analytic purposes, CIM is 
supposed to provide useful and meaningful information upon which real policy 
analysis can be based. The student of policy is not seeking stimulation, or 
indeed education in how properly to think about the future but rather to find 
concrete answers to complex questions about the future.) 

I have found that the problems with the methodology which need to be 
changed or replaced before it can provide a useful tool for policy analysis 
can be partitioned into two categories: 

(1) underlying assumptions about (a) the meaning of the inputs, 

(b) the ability of experts to perceive and to a rat-ly 
describe t' °.ir perceptions of the inputs, and (c; the ability 
of mathematics to handle th inputs; 

(2) those problems concerned with the present model's inability to 
do those things that it says it does. These include, for 
example, invalid mathematical formulas as well as questionable 
use of simulation techniques. 

Though I have thought more about (1), I have written primarily about (2), 
because such problems are easier to specify and, indeed, to deal with. I 
do try to indicate some major questions inherent in (1) in the last section, 
but I must defer deeper discussion of these subjects until a later time. 

N.B. After the first draft of this paper was completed I became 
aware of the Institute for the Future's Report R-12 by Gordon, et. al. 

Their report addresses many of the problems discussed in this paper, but 
in my opinion most remain unresolved. 
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PROBLEMS IN THE CIM MODEL 



The main text of the paper is written under the assumption that the 
reader is familiar with the cross-impact matrix methodology. Those who 
need to familiarize, or re-familiarize themselves with CIM, should read 
Chapter I of the IFF Report on "The Use of Cross-Impact 1 atrices for Fore- 
casting and Planning"* and/or Larry Hudson’s paper "Uses of the Cross-Impact 
Matrix in Exploring Alternatives for the Future"**. The version of CIM 
referred to in this paper as "the present version" is that which is explained 
in taose two documents. 



A. The Mathematical Model 



In this section I discuss some aspects of the formula*** which seem 
to fail to reflect reality or which are inconsistent in some other way. 



Reproduced in Appendix B. 

Lawrence Hudson, Uses of the Cross-Impact Matrix in Exploring Alternatives 
for the Future , EPRC Working Draft, December, 1969. 






V 



B 



- P B + AP B “ P B + V 1 ~ V S AB ■-*- 



where : 



= "initial probability" (for event B) ; probability of event B 
occurring by time t (see below) without consideration of the 
occurrence or non-occurrence of event A. 



S.^g = "impact" of event A on event B. 

t = "time horizon"; time in future for which the probabilities are being 
estimated . 



t^ = year of 50% probability (yfp) of event A; i.e., the first year by 

which it is thought that the event has equal chance of occurring or 
not occurring. 

Pg' = "adjusted probability" (for event B) , assuming the occurrence of 
event A. 



t-t 



Note: AP b = Y - P B “ P B ( 1 - P B )S AB 



V 



A >5030U 



1. The Time Factor Function. 

11 " t A 

The expression under discussion here is — , or time factor . 

The present version of CIM seems to assume that the closer the yfp, t^, of an 

event is to the time horizon t, the weaker will be its effect upon the other 
events (See Figure 1)*. If the yfp is year 0, the effect is complete (the 




Figure 1 . Time factor as a decreasing linear function of t.. 



t “ t 

time factor A = t-0 = 1) ; if it is half way between year 0 and the time 

t t 

horizon, the effect is half of this ( *“ ^A = t-(t/2) = 1/2); if the yfp is 

t t 



k 

Since an event which occurs late in the time period has less time for its 
influence to spread, it is presumed to have less impact on other events. 
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t 



= 0) . The assumption seems to be 



the time horizon, the effect is zero 




that in estimating the change in probability , we are assuming that the event 
has the full time between year zero and the time horizon to effect the other 
events, and that by naming yfp (t ) , we are saying that, on the average, the 

event will happen in year instead of year 0, and- therefore, the effect 

will need to be decreased somewhat, depending on how close t is to the 

time horizon. 



In fact, however, it is possible that the dependence may not decrease with 
respect to time. For instance, event A might be so intimately connected with 
event B (B might be a highly desired goal whose attainment depends chiefly on 
the occurrence of A) that there is a high probability that B will occur immediately 
after A. It matters not whether A occurs close to year 0 or late in the time 
period; B will be impacted upon in the same way. In this case, the dependence 
of B on A is not decreasing but is essentially constant with respect to time. 

This sort of interaction might be represented by some function such as the one 
in figure 2a. 



The time factor also assumes that the dependence between events is linear 

with respect to time. The question of whether the. dependence should be linear 

has not been addressed. Linearity was chosen because it is simple and it makes 

some sense, but the following examples show that it might make no sense. For 

example, suppose there were (perhaps unstated) linkages between two events A 

and B (a) which took time to occur, (b) which would occur with high probability 

if A occurred, but (c) which would have to first occur before A could impact 

on B. In this case the effect on A on AP would be constant with respect 

J3 

to time, until that point in time after which the intervening events would 
net have time to occur before the time horizon, and hence A would not have time 
to influence B. The time factor function might look like the one in figure 2b. 
A similar example is the case where A' s influence on B might be vitiated beyond 
a certain time due to some sort of "deadline" event, such as an election, 
whose occurrence was exactly predictable. 




Figure 2 . Alternate time factor functions when (a) there is no 
time lag between occurrence of A and its impact on B, 
(b) there is a definite time lag, and (c) impact of 
A on B increases with time. 



As a final example, it is conceivable that the impact of A on B might 
even increase as the time of occurrence of A gets later, due to changing 
circumstances surrounding the events. For instance, if current studies on 
"total mastery learning" had occurred ten years ago, when the "spiral curriculum" 
was just coming into vogue and there was little thought about individualized 
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ins “action, these studies would not have had nearly the impact they promise 
to h’ve today. The time factor function might look like the one in figure 2c. 

Since, as the above examples show, it is not always the case that any 
given linear function, and in particular a decreasing one, represents the depend- 
ence of AP on time, it seems useful to consider the times of occurrence of events 

D 

only if one looks at the changing probability as a function of time, each 
individual case determining its own function. This means that you would have 
to describe such a function individually for every non-zero impact in the 

matrix. 



Initial Probabilities Less Than 0.5. 

Another problem with the present linear function occurs when the impacting 

event (event A) has initial probability less than .5, so that its yfp must 

faU beyond the time horizon, and the time factor becomes negative 

(*" t A < 0 if t A > t). This causes AP^ to change sign, which is like saying 
& ii 

that if A's occurrence "enhances" the probability of B (i.e., if the cell 
containing A's impact on B has a positive entry) and if A has initial probability 
less than .5, the actual occurrence of A will cause the probability of B to 
decrease. * 



This paradoxical situation could be gotten around in at least 4 ways: 

(1) set the time factor to zero whenever it is less than zero 
(Figure 3a) ; 



(2) use another non- increasing function (necessarily nonlinear) , 
which would approach zero only asymptotically, such as 





if t. * 0 
A 

if t. < 0 
A 



where a is a constant (Figure 3b) ; 



It could be argued that in some cases this is desired, but such cases 
probably imply a more complex set of interactions than CIM can handle. 
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(3) as discussed above, describe a function separately for 
every impact; 

(4) don't consider the time dependence at all. 





0 1 i 1 1 1— 

-0.5* 0 .5* t 1.5* 

(b) 



Figure 3a . 



Figure 3b . 



Linear time factor function which does not change for 
t^ < 0, and t^ > t. 



Time factor function which approaches zero asymptotically 
as t^ increases, does not change for t^ < 0. 



3. Year of 50% Probability (yfp) Less than 0 or Greater than 2t. 

A further problem, related to the last one, occurs when the yfp for the 
impacting event (event A) occurs before time zero or after time 2t* (t is time 
horizon) , assuming in the latter case that the problem mentioned above has been 



For example, suppose that year zero were 1970 and the time horizon were 1980 
(i.e., t = 10). If a respondent felt that an event A had in alx probability 
already occurred, then the yfp for A (t^) would be less than 0. If he felt that 

A was not likely to occur before 1995, then the yfp for A would be greater 
than 2t . 




11 . 




solved. In either case, the time factor is greater than 1 in absolute value 
(Figure 4). This means that AP^ can be made as large as desired merely by 

adjusting the yfp of A. For example, let event A have yfp = t = -30, time 

A 

horizon t = 10, and impact S = .8 on B, whose initial probability is 
= 0.9. If A occurs, the impact of A on B gives 

JJ 

V = p b + ^a W P B> (1 -V 

= .9 + 10 + 30 x .8(.9) (.1) 

10 

= .9 + 4 x (.8) (.09) 

= .9 + .288 I 

= 1.188 



N 
O 
« n 

< 




(TIME HORIZON) 
YFP ( = t A ) 



Figure 4 . Time factor function. Function value falls out of the 
range -1 to +1 in shaded areas. 
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Hence, it is possible with the present program that if A and B are events such 
that A has non-zero impact on B, the adjusted probability for B may become 
greater than 1 or less than 0, depending only on the yfp for A « This has 
three interrelated disturbing aspects. 

First: probabilities greater than 1 and less than 0 should not occur.* 

The fact that the given formula yields such probabilities might well lead one 
to doubt the reliability of the formula. Indeed, this situation is^ patched 
up in the present CIM program by resetting the adjusted probability to 1 
whenever the formula yields an adjusted probability greater than 1, and to 
zero when the formula yields an adjusted probability less than zero. But 
why a formula which clearly yields unreasonable results at some places on 
the spectrum is considered reliable after alteration at those places (and 
without alteration elsewhere) remains to be justified. 



Second: the set of situations in which the occurrence of an event B 

can be made certain or impossible by the occurrence of another event A is 
theoretically infinite in the program,** contrary to a reasonable expecta- 
tion that such a set would be very small (if indeed it were not empty). 



Third (and this is due to the linearity of the time factor function) : 
the formula tells us that no matter how small the effect of event A on 
event B, if A occurs early enough then B's occurrence is forced, while if 
A occurs late enough, B's occurrence is made impossible.** Common sense 
tells us that this is unreasonable. 



Allowing probabilities greater than 1 or less than zero is like saying an 
event is more likely than certain, or less likely than impossible. 






Depending only on t (= yfp of A), for t < t 

A A 



t 





yields 

t > t + 
A 



the certainty of B's 



S AB (1 V 



yields 



occurrence, and 

the impossibility of B's occurrence. 
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Again figures 3a and 3b offer possible solutions to these problems. 



4. Constraints on Change in Probability (AP). 

Ironically, the above aspect of the model gives the user a certain 
flexibility which, due to the quadratic nature of the impacting formulation, 
he would not otherwise have. Figure 5 shows that in a "normal" situation* 




Figure 5 . Effect of impact factors** (time factor assumed to be 1) . 

Shaded area represents the amount the initial probability 
can change in "normal" circumstances. The three lines 
represent the maximum impacted probabilities for 
S = 1, 0, -1. 

the amount of impact that one event can have on another is relatively 
small, a maxim AP*** of an event (due to the impact of one other event) 



By a "normal" situation I mean one in which the yfp of A is between 0 and 2 

Adapted from A Case Study Using Forecasting as a Decision-Making Aid , 
by S. Enzer. p. 24. 

As before, AP is that part of the formula which is added to or subtracted 
from the initial probability to get P . Thus it is the 

"P(1-P)S t ~ t A " part. 



t 



being .25, and this only when the initial probability is .5. This means 
that all the area above the S = +1.0 curve and below the S = -1.0 curve is 
unattainable in a single impact, under "normal" cii cumstances . It, 
therefore, means that necessary or sufficient conditions cannot be represented 
in the matrix. * I.e., if the occurrence of event A were sufficient, in 

the strict sense, to force the occurrence of event B, this could not be 
represented by a "normal" matrix. Of course, the restriction is much stronger 
than just not allowing necessity and sufficiency. In fact, in the present 
model the amount of change which is not possible under "normal" circumstances 
is greater than the amount of change that i£ possible (the unshaded area in 
Figure 5 is greater than the shaded area) . What might be preferable would 
be for the upward limiting curve (corresponding to S = +1.0) to be made as 
close to the upper boundary as the user desires (see Figure 6), and similarly 
for the downward limiting curve.** 




INITIAL PROBABILITY 



Figure 6 . A more desirable range of effect of cross-impact factors. 



The "priority level" parameter was introduced to take care of necessity, 
but it is hardly satisfactory as it stands since it requires either that 
all events, except the one that is necessary, depend on the necessary 
event, or that all events except the one which depends on Lhe necessary 
event be necessary events. (See Hudson paper.) 

In S.Enzer, Delphi and Cross-Impact Techniques , a method is suggested for 
doing this by scrapping the quadratic formula and using "changes in 
likelihood." The computation procedure is not spelled out in the article. 
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The quirk in the model due to the time factor (last section) allows 

the user to force the total impact to be greater than the change shown by 

the limiting curves in Figure 5, simply by letting the pdo be either less 

than 0 or greater than 2t. For example, if the initial probability of event 

B is .5 and S AT1 = 1.0, the occurrence of A could be made to raise the 

AB 

probability of B by .5 (to 1.0) by setting the pdo of A to -t. 

I don't consider this a very good way of inducing a greater control 
over AP for several reasons: 

(1) it perverts the meaning of yfp; 

(2) all events are affected, so that one could not use it to induce 

a "strong" impact on one event without inducing a "strong" impact 
on all events; 

(3) it further complicates a fairly simple idea; 

(4) as stated above, it gives infinite domain to those impacts which 
force certainty or impossibility. 

5. Importance of Yfp of Event B. 

Another problem with the present CIM program is that although common 
sense tells us that the impact of A on B is dependent not only on the 
yfp of A but also on that of B, in the present program the yfp of B is 
disregarded.* For suppose that A has given impact on B, B has given 
initial probability and A has given, say early, yfp. Either of two 
situations could occur: either the yfp for B could be early, near A's, 

or it could be much later. Surely if the effect of the yfp of A on the 



Although we may assume that in general the yfp for an event with initial 
probability .8 falls before one with initial probability .6, we have no 
precise notion of when either yfp occurs. Indeed there is no reason to 
assume that two events with the same initial probability will have the 
same (or even approximately the same) yfp . 
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occurrence of B is indeed time dependent, these two situations ought not 
to be considered identical as they are in the present CIM. 

So far I have done little actual work with the problems of the effects 
of yfp on the impacts, but, recognizing the trc_ble one can get into with 
the straight linear dependence, I have rewritten our programs so that they 
optionally either do not consider the yfp (set time factor to 1) , or they 
consider any event with initial probability less than .5 to have zero 
impact on all other events. Neither method is totally satisfactory, though 
we are leaning toward more use of the former than the latter for the simple 
reason that events with low initial probability very often do come into 
cons ideration. 

B. Order of Occurrence (ooo) of Events 
1. Importance of ooo. 

Except for priority levels, events are assumed to occur in a random 
order. E.g. in game 1, we may have the order A, B, C and in game 2, order 
B, A, C. Hence, in game 1 event A impacts on B and C but is not impacted 

upon. B impacts on C only and is impacted upon by A. C is impacted upon 

by A first, then by the changed B. In game 2, B impacts on A and C and 
is not impacted upon. A impacts on C and is impacted on by B. C is 
impacted on by B first, then by the changed A. Obviously the order in 

which the events are assumed to occur has some influence on the final 

probabilities of a given game. In game 1, for example, A is not impacted 

on at all. In game 2, it is by B. If the impact of B on A causes 

(’’probability of A”) to rise by 20%, then A would occur 20% more times 

playing the game 2 than game 1. Event C, which occurs last in both cases, 

is affected differently. If we assume (for simplicity) that B is not 
changed by A, then C is impacted on 20% more times by A in game 2 than 
in game 1 . 



17 




2S 



Clearly, the order in which events are handled by the program is very 
important . 

2. Present Program and Assumptions. 

The way the present program handles this problem is to play many 
games, generating a new random permutation of order N (where N is the 
number of events) for each game. There are some minor changes to this approach 
which, I think, will improve the outcome and perhaps cut down the number of 
games . 

Some assumptions on which this procedure seems to be based are: 

(1) that each permutation (within a priority level) is equally 
probable. That is, all events have equal probability of occurring 
at any given position in the order of events; 

(2) that y playing a large number of games the permutations will 
balan a out around some average, i.e., that in the long run, 
the random generation of permutations will yieJd a set of 
permutations in which all different kinds of permutations occur 
with about the same likelihood. 

Based on these assumptions (which can and will be questioned later) 

I have made some further assumptions: 

(3) that, given (1), a complete game would involve each permutation 
of order N occurring a large and equal number of times. For 
instance, in the 3-event example, we would want each permutation 
(ABC, BCA, CAB, ACB, CBA, BAG) to occur enough times (say m) 

to yield accurate (within prescribed confidence limits) final 
probabilities. If a certain permutation were left out, the 
influence of the possible future represented by that order of 
events would be lost. Hence, in a 3-event matrix this would 
yield m x (3!) =6m games,* with each permutation occurring m times. 

the number of permutations of order 3 is 3! = 3*2*1 = 6. The number tof permuta- 
tions of order N is N! = N* (N-l) *(N-2) .. .3*2*1. 
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Such a "complete" game becomes unwieldy as N increases. For 
a ten event matrix, this formula would require m x 10 ! = 3,628,800 x 
games. In actual practice, fairly good confidence limits can be 
achieved by playing much fewer games.* 

(4) that, given (1), no permutation should occur more than once 
unless more than Nl games are played; 

(5) that, given (2), for every game in which an event occurs "early" 
in a permutation (hence impacts on most other events but is 
impacted on by few) there ought to be a "complementary" game 

in which the event occurs "late" in the permutation (hence is 
impacted on by most events but only impacts on a few); 

(6) eve -v event should impacc upon every other event as many times 
as z.'i is impacted upon by every other event. 

If the permutations are chosen totally at random, the more games that 
are played, obviously the more likely it is that these assumed requirements 
will be satisfied within prescribed limits. One would suspect that one 
way of reducing the variance (or, alternatively, the number of games) 
would be to require the random permutations to satisfy as many of the above 
assumptions as possible. It turns out that by considering cyclic permuta- 
tion groups of order N, one can do this fairly easily. 



I define a cyclic permutation group (CPG) of order N as a group of 
permutations of order N all of whose elements are "cyclically equivalent."** 



Richard Rochberg, "Convergence and Variability Because of Random Numbers 
in Cross-Impact Analysis." Also Howard Johnson, "Some Computational Aspect 
of Cross- Impact Matrix Forecasting." 

I define cyclical equivalence: A set of permutations which give a circular 

arrangement, as in the figure on the right, can be 
used to generate the cyclically equivalent per- 
mutations. Any other permutation that can be 
got by rotating the entire circle is cyclically 
equivalent. If the figure above represents the 
permutation 12345, its cyclically equivalent forms 
are (12345), (23451), (34512), (45123), and (51234). 




Every such group contains N distinct permutations. Therefore, since there 

are N! permutations of order N, and since every permutation is a member or 

Nl 

some cyclic permutation group, there must be-^ = (N-l)! unique cyclic 

permutation groups of order N for every permutation of order N. 

No two different CPG’s can have any permutation in common since if they 
did each element of one would be cyclically equivalent to each element of 
the other, and they would be the same. Hence, if one chooses random permuta- 
tions from different CPG's for each game, there is no chance of repetition 
up to (N-l)! games. If less than (N-l)! games are :layed, assumption (4) 
is satisfied. 

A simple way to satisfy assumptions (5) and (<: ) would be to do the 
following: for every game in which a certain permutation is used, play 

another game in which the order of the events is reversed. For example, 
if the permutation 6 1 5 3 2 4 is used in one game , another game would use 
the permutation 423516. 

Of course, if the permutations were generated in a completely random 
fashion, eventually one would be reasonably close to satisfying assumptions 
(4) (5) and (6), but if one could systematically see that they are satisfied 
throughout the running of the program, variance due to randomness could be 
reduced and fewer games would be necessary. 

I have developed a procedure for doing this in a semi -random fashion. 
This procedure is based on an algorithm* which generates from a given 
permutation all the equivalent permutations (in the same CPG) , then 
automatically generates a "next" CPG of permutations, and so on until 



See Appendix C. 
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have been generated. A sample run, with the permutation 1 2 3 4 of 
order 4 as input, would look like: 1234 

2 3 4 1 

3 4 12 

4 12 3 

2 3 14 

3 14 2 

. (24 rows) 

• 

• 

3 2 14 
2 13 4 
14 3 2 

4 3 2 1 

A nice feature of the order in which the permutations are generated 
is that the ith permutation in the list is the mirror image, or reverse, 
of the (24-i)th (in general, the (Nl-i)th) permutation. Hence, the reverses 
of the permutations generated in the first half of the permutation list 
(therefore, of the CPG's also) are in the second half of the list. 

Based on this information, the procedure goes as follows: 

G 

(1) generate a set of — random integers R^, where G is the 

G 

number of games to be played, and 1 £ R„ £ — 

'L Z 

(2) randomly generate a permutation, whose CPG will be the "seed" 
CPG, or C?G 1 

(3) set index counter i = 1. 

(4) "rotate" CPG^, R^ times. This gives the permutation P^^ ^ to 
used in the (2i-l)th game. 
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(5) reverse P_. . to give the oerrutation P_. to be used in the 
2i-i & ‘ 2i 

(2i)th game, (If i < (N-l) ! ".either nor ^2i Can ^ ave 



been selected previously!) 



(6) use algorithm to genera's nest CPG. 

(7) if G permutations have not been determined, return to (4). 
Otherwise, process is finished™ 



Symbolically : 



Oi 

«0 

o 

to 

< 




If G<N! assumption 
(6), and the randomness 
define "early" as among 



(4) is satisfied; step (5) 
of the procedure satisfies 
the first third to occur , 



satisfies assumption 
assumption (5) if we 
and play enough games.* 



We have found that for a 10 event matrix, 30 games is sufficient to ensure 
95% confidence that a given event will occur in the first third at least 
once. For a 30 event matrix 100 games are required, and the number of 
games rises with the number of events. 



3. Omission of Monte Carlo Procedure. 



I refer to assumption (3) in which . ' jlete game involves m x n! 
games, where m is sufficiently large to :•<=. ra variance (in the interaction 
of events due to random tossing of weigh:.- •_ coins) to an acceptable level. 

I argue that at this point in the playing . a game a fairly simple analytic 
process yields the exact final probabil chat the random coin— tossing 

procedure converges to. Hence, m is recr so 1 game and a complete game 

is reduced in length by a factor of m, t games. 

Consider first a simple 3 event mar.: : events A, B and C) with 

initial probabilities P & , P g and V Q> and ; — :=ts S AB , S^, S^. S^, S CA , 

S . Since there are 6 possible ooo’s, az ceal run would involve 6 x m 

games, or m games for each ooo, where m shx:_ld be sufficiently large to 
yield accurate final probabilities. To show what we want to show we need 
only consider a single ooo, say ABC. (A considered first, then B, then C) . 

In each of the m games using ABC as the ooo, there are only two possible 

outcomes for event A: either A occurs (1), or A fails to occur (0). In the 

case of 1 we calculate P * ■* P_ + P (1-P_)S A _ x 1. In the case of 0, 

■D Jd B AS 

Pg' = Pg + Pg(l-Pg)S^g x 0 = Pg. The probability of 1 in any game (say 

the Kth game) is P^, the probability of 0 in 1-P . 

Now let us look at the method used t he CIM to arrive at final 

probabilities (in this case, for the m ganas only). For each game a sample 

value, or random variate, (0 or 1) is generated according to the distribution 

described above. If P. = .7, then about 70% of the time the variate 

A 

would be 1, the other 30% would be zero. (If m were 1000, wa would expect 
about 700 l f s and about 300 0*s.) To figure the final probability of A 
the program calculates the arithmetic mean (average) of the m variates. 

If our method of generating random varia s were correct, we would expect 
that mean to equa 1 approximately P. . Jane ^ ^ as far as the generation of a 

A 
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final probability for A is concerned, we had the best estimate possible 
before we began, namely P A . 



Now consider P ' . We said above that P ' = P._ whenever A was assumed 
b a d 

not to occur, otherwise P ' = P + P (1-P )S A _. If m games were played, 

B B B B AB 

obviously the increment P_(l-P t3 )S A would be added about P (70% in the 

above example) of the time. Now, the final probability for B is found 
much the same way it is found for A, namely by generating random variates 
according to the distribution given by P^' . For example, suppose P^ = .7 

and P,,' = .5 when A does not occur and P^,’ = .5 + .2 when A does occur. 

B B 

Let us also suppose 1000 games were played. In about 300 games P^’ = .5 
would be used to generate random variates and in about 700 games P^’ = .5 + .2 
would be used. Hence we would expect to turn up a 1 about 

(.5 x 3G0) + (.5 + .2) x 700 = (.5 x 1000) + (.2 x 700) = 500 + 140 « 640 times, 

yielding a final probability for B, P^, of .64. Obviously the increment 

at 

(corresponding to occurrence of A) is added to P^. that proportion of the 

B 

time that A occurs, so that P_,_ = P_. + P. x P„ x (1-P_)S._, (in the example 

Bi? BAB B AB 

P^.-, = .5 + (.7 x .2) = .64) is the best estimate of the final probability 
Br 

of B and the one that the Monte Carlo procedure would converge to for m 
sufficiently large. 

For event C the final probability, P , would be similarly arrived at, 

CF 

though the computation is more involved since C is acted upon by both A 
and B. The reader may war-t to verify that 

P CF = V + V X ^ S BC X P BF 



where P^ = P c + P c x (1-P C ) x s ^ c x P A - 





For a larger matrix, a similar procedure as the one described here would 
be used.* Hence we have a way of assessing the exact impact of one event on another 
without using Monte Carlo methods , and we are able to reduce the number 
of necessary games by a considerable amount. Indeed, it turns out that most 
of the run-to-run variance is due to the random decision on whether or not 
events occur, and not on the order of occurrence, so the above procedure 
serves to eliminate most of the variance. 

I have made several test runs using this technique together with the 
technique described above for generating order of occurrence, and tentatively 
can conclude that for a ten-event matrix ten games using these methods give 
as accurate results as 1000 games using the previous technique. 

4. Alternative Ways for Generating ooo. 

Assumption (1) is "that each permutation within a priority level is 
equally probable." The use of priority levels is itself questionable since, 
if used as it is described in the Hudson paper, it does not allow for any- 
thing but the simplest and most unlikely notions of priority, i.e., those 
in which one set of events must have priority over all other events. It 
does not allow, for instance, for the case in which one event has priority 
over another event and all other events are not involved in any way in 
priorities. IFF seems to have recognized this long ago, since they do 
not seem to use it in any of their programs. 

But back to the question of ail events having equal probability of 
occurring at any given position in the order of events. It would seem an 
obvious refinement of the method to assume that certain permutations of 
order of occurrence would be more likely than others. For example, an 
event with year 1 as its yfp would be assumed to occur before another event 
with year 10 as its yfp more often than not. Hence more games with permutations 
having the first event before the second should be played. 

A 

G. Kutsch, Mathematical Formulation of CIM Process as of November 1970 , gives 
a description of the computational process involved. 
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One way to accomplish this (which we have not tried), would be to 
consider the distribution of yfp's for each event as estimated by the 
P ar t^- C i-P an t:s of the study. An example of such a distribution is the inter- 
quartile polygon commonly used in delphi studies. Figure 7 shows a time 
with (a) polygonal and (b) continuous distributions, rather than 
points on the line, representing the yfp's. Note that sometimes they overlap 
and sometimes not. One could take a random sample from each distribution to 
get a sample set of "dates of occurrence," which in turn would determine the 
sample order of occurrence for a game. 



Alternately, if the number of participants were small, one might rather 
play one game for each participant, using the yfp's of that participant to 
determine the order of occurrence for the corresponding game. Or similarly, 
one could play several games for each participant, sampling from some simple 
distribution about each yfp. 

I , 

0 t 

(a) 




<b) 

Figure 7 . Time line with (a) polygonal distributions and 

(b) continuous distributions, rather than single points, 
to represent yfp’s. 

I 

The above described alternative methods of generating ooo ("order of 
occurrence | ) permutations accomplish two things : they generate more likely 

ooo s of events, and they force the necessary number of games (to achieve 
a given confidence level) to be dependent upon the amount of agreement among 
the participants. If the participants agree completely, then perhaps only 
one game is necessary. Otherwise, the number of games could be greater than 
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the number of participants. In any case, I would expect the number of games 
to be less than by previous methods. Furthermore, if one of the above 
methods did prove to be superior, assumptions 4, 5, and 6 would no longer 
be valid, greatly simplifying the concepts underlying the method. 

C . Specifying Input Information 

1. Assumptions Constraining the Formula. 

There are two assumptions placed on the impacting function which force 
it to be at least quadratic: if = 0, then P ' = 0, and if 

P^ = 1, then P^' = 1, If we retain these assumptions, we are faced with 

the problem (described in II. A. 4) of an unreasonable limit to the maximum 
change in probability which could result from the occurrence of a single 
event. Perhaps one way to get around this problem is to keep the two assump- 
tions and see how the concept of Impact might be redefined so as to allow 
a greater range of change in probability due to the occurrence of a single 
event . 

2. The Definition of "Impact." 

It is never made clear exactly what is meant by a "strong" or "weak" 
impact of one event on another. The user is not expected to try to under- 
stand, except relative to his other impact estimates, how one event is 

really affecting another when he says it has a strong impact.* It would be 
likely that different users have different assumptions about what this 
does mean. One user might, for instance, think that an impact of +1 would 
cause the impacted event to be 100% more likely, while another might think 
it would cause the final probability of the impacted event to be equal to 1. 

The cctctp: of "impact" is rigorously treated from a definitional angle, 
but without ever getting into the differences in meaning between different 
numerical values, in Gordon and Hayward, On Correlations Betx-yeen Events . 

The Rochberg, et. al., paper begins to ask about these differences by 
suggesting "sensitivity analysis," but this is of no value to the user 
for his initial estimations. 



Most users would probably expect an impact to have just as great an effect 
on a low or high probability event as on an event with probability .5, 
while others, who perhaps understand the formula, might understand that the 
strength of the impact is very dependent upon the initial probability 
estimate of the impacted event. 

What is needed is a careful definition of the meaning of "strength 
of connection," or impact, one which is consistent with the formulas and 
which can be clearly understood by the user. Otherwise, the CIM is nothing 
other than a black box both to the participant, who doesn't understand what 
is being done with his inputs to arrive at a set of final probabilities, and 
to the formulator of the matrix, who does not understand what the inputs 
mean to the participant. (The problem of the meaning of strength has not 
escaped the concern of the IFF, and they have been toying with several 
alternatives, though they seem to have continued to use the standard 
formula in working with other groups.) One would hope that the impact would 
be defined somehow in terns of its use in the formula. Some straightforward 
examples : 



a. In each cell of the matrix are entered two numbers, (1) the 

probability of B occurring if A occurs, and (2) the probability of 
B occurring of A fails to occur;* 



* 

The initial probability of B is supposed to be arrived at without consideration 
of A's occurrence or non-occurrence, but as we can see from the following 
example this is not always practical or indeed possible. For suppose that 
event B were the winning of the 1971 pennant by the San Francisco Giants. 

And suppose the participant felt that the chances were good (say Pg = .8) 

that the Giants would win — but his prediction was based implicitly on his 

supposition (say he is 90% sure) that Willie Mays will play for most of the 

rest of the 1971 season (event A): if he knew that Willie Mays would (or 

would not) play then his selection of P might be more like .9 (or .3). 

B 

This kind of situation cannot be dealt with reasonably by the CIM in its 
present form. 
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b. Again, two numbers in each cell, (1) the change in probability 

of B if A occurs, (2) the change in probability of B if A fails to 
occur; 

c. One number, the change in the probability of B given that A 
will occur with the expected probability; 

d. On a linear scale, the proportion of the difference between P_ 

J3 

and 1 (or 0) that the probability will change (1) if event A 
occurs, and (2) if A fails to occur. 

Undoubtedly many more formulas could be suggested for defining a 
strength of connection. The important points to keep in mind are that the 
user must understand what he means when he defines a strength of connection, 
he must understand what that number does in the formula, and he must 
understand how its role changes when the matrix changes. Also, the 
strength of connection must give him complete power to alter initial 
probabilities as much as he thinks they should be altered. If he does 
not have this power, there is really no reason to set up the matrix in terms 
of probabilities. 

3. The Meaning of Probability. 

Certainly one of the major drawbacks of a technique such as cross 
impacting is that the procedure tends to keep the user in the dark about 
what the machine is really doing with his information, and he may not 
even understand what the information he is giving really means. This is 
especially so when the user is asked to give an estimate of "initial 
likelihood of occurrence," or initial probability. It is absolutely 
essential that the user knows exactly what he mea-is by probability, since 
his estimates are to be treated in a very strict, mathematical way, and if 
the meaning of the estimates is to be fuzzy, then we can only expect the output 
of the treatment to be even fuzzier. 
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The question of subjective or personal probability is a very important 
one to much of the futures methodology, and one whicJ" we should give careful 
attention to, but not one I am prepared to get into in this part of this 
paper, since this section is primarily about the tactical procedures involved 
in the CIM. 

D . Summary of the Problems 

At least one thing should be obvious from the preceding discussions 
of various components of the CIM methodology. There are so many questionable 
areas which require either further refinement or complete conceptual change 
that it is foolish to think of the method as producing anything but the 
roughest estimates. 

Initial probabilities are very personal, very biased estimates c c 
likelihood, and only in the very loosest statistical sense can they be 
called probabilities. At best, they give us some idea about how people 
perceive the r elative likelihoods of each of a set of events. The set of 
events itself represents a personal attempt to describe an environment in 
terms of all important events and how they interact. Which events are 
implied or assumed without ever being stated is an open question, and 
probably different for every uEer. We have to take on faith the partici- 
pant's ability not only to perceive correctly the complex interconnections 
in the environment, but also to make explicit his perceptions, although in 
fact he may not even be aware of some of them. 

The yfp has all the disadvantages of the initial probability which, 
added to its very influential role in the methodology makes it even more 
in need of better definition and understanding. Because of the confusion 
about how to handle the time path of probability, as a function of yfp, 
we usually do not even use it in our tests. There are indications that 
IFF also does not employ it in some cases. 
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The impacting formula as it now stands, with the factors P and 1-P, 
forP less than one and all other factors less than one, forces undue 
restrictions on the amount of influence one event can have on another. 

Priority levels, as defined in the original nu 'hod, just do not make 
sense in most cases, so they have been dropped altogether and in their 
place I suggest a parameter for handling necessity and sufficiency for each 
pair of events. 

I feel that there are far too many questionable factors in the method 
to generate any kind of meaningful probability information. I would feel 
much better about the method if it did not pretend to give results in terms 
of 'final probabilities' or 'changes in probability,' but simply gave the 
output as a kind of final score to a game, without any implication that one 
could know the real likelihood of occurrence of a set of events from using 
the CIM any better than he knew it before using the CIM. 
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THE USE OF CIM 



A. The Heuristic Value 

It has been suggested that the most valuable part of the CIM method 
is the determining of the inputs. This is the stage when the participants 
are forced to examine very carefully the events which are most important 
to consider, all the complex interactions among the events, and the likeli- 
hood and importance of each event within the whole environment. The next 
stage has the computer doing all sorts of questionable manipulations of the 
carefully thought out input data, and generating sets of output which force 
the participants to rethink their original assumptions and hopefully sharpen 
their understanding of the complexities of the environment. The important 
contribution of the method is not that this latter stage generates a more 
accurate picture of the environment, but that participation in the CIM 
stimulates improved perceptions about the environment. 

For heuristic uses, it may not matter that the method does not work 
properly as long as the people using it believe that it works correctly. 

That is, the goc 1 is usually not to find the right answers, but to stimulate 
the "right" kind of approach to futures, to enhance the "futures perspective," 
as it were, of the participants. In fact, one wonders whether it might 
not be better that the technique works improperly, at least some of the 
time, making it more likely that the participants will be impressed by how 
different their perceptions of the future can be from the "correct" percep- 
tions.* 

It may not matter, for heuristic purposes, whether or not the method 
provides us with new knowledge, as long as we think it does. CIM may tell 
us only things we already knew, but if it tells us those things in such 
a form that we think we didn* t know them, then we are impressed, and 

__ _ — — 

I have seen this very phenomenon work very well in several CIM exercises. 

Of course, it always worked best when the leader did not himself realize 
that the cross impact analysis was incorrect. 

O 
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perhaps stimulated more towards the "right" kind of thinking. In this 
case we see v, °w analytic techniques can help us correct our own "mistaken" 
perceptions . 

It may also not matter that the implied philosophical p^stular.es 
upon which the method is based have never been justified, as long as 
the participants accept them and the method as valid, and hence "learn" 
that the discipline of futures research has been generating numerous 
analytic techniques for studying the future. An aim of an heuristic 
futures exercise is, after all, to help break the mental set of the 
participant who has not developed a futures perspective. 

In short, there may be strong justification for the use of the CIM 
as an heuristic tool, and until it can be demonstrated that the method gives 
much better than "ball park" answers, it should not be used for anything else. 

Furthermore, if the method is merely an heuristic one, I suggest 
there are much simpler procedures for accomplishing the same thing, without 
the need for questionable formulas, fancy Monte-Carlo techniques, or perhaps 
even a computer. I outline one such procedure now. 

B. Simple Alternative Procedure 

Generate the same input information as for the CIM (except for the 
priority levels and yfp's). For each column, multiply the contents of 
each cell in the column by the corresponding initial probability and add 
up all results in the column. Mathematically, this means taking the 
cross-product of the vector of initial probabilities and the matrix of 
impacts. The result is a vector of positive and negative numbers, with 
each element of the vector giving a 'total impact' on the corresponding 
event of all other events. The meaning of the results is quite simple. 

It is not a probability, nor a probability change. It is a final score, 
the result of giving events which are more likely more weight, and giving 
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the heavier impacts more weight. The events which have the highest 
positive or lowest negative scores are those which are influenced most by 
the totality of interactions. 

Now, if one is interested in those events which are more influential, 
one could generate another vector by summing the impacts in each row and 
multiplying that sum by the initial probability of the corresponding event. 
This would give the "total impact" of each event on all the other events. 
Sensitivity analysis could easily be done from outputs by minor adjustments 
to the above procedures. Importance analysis is also quite easy: one need 

only look at the relative sizes of scores in the two output vectors. For 
example, an event which had low relative scores in both vectors can be 
assumed neither to have great influence on, nor be greatly influenced by, 
the other events. 

I have played with this procedure, using the inputs from IFF’s case 
study done at the college of Europe and Bruges in 1969*, and found that this 
method provided the same information as they were able to garner by the CIM. 
The advantage of the simpler system is that it is not a black box to the 
participants and that it does not produce results in a form that is likely 
to be misconstrued. 



Selwyn Enzer, A Case Study Using Forecasting as a Decision-Making Aid , 
Institute for the Future Report WP-2, December, 1969. 
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IV 



IMPLICATIONS FOR FUTURES METHODOLOGY 



I would now like to discuss the content of this paper in light of 

the whole area of futures methodology. I hs^e said that I see CIM 
useful as a pedagogical device for raising questions about likelihood 
and order of occurrence of sets of events, about interrelationships among 
events, and about the consequences of policy actions which could affect 
the values of these parameters . I have serious doubts about whether CIM 
could ever be used as a reliable decision-making tool. 

Many of the limitations of CIM that we have discussed in this paper 
arise generally in the area of futures methodology, and I think it Is 
important to examine other research methods in the light of these limitations. 
Unless we can get a more analytic command of the concepts underlying the 
techniques we use*, the future of futures research may be limited to the 
development of purely heuristic methods. I will illustrate the kinds of 
problems I am thinking of by briefly discussing three problem areas. 

1. Simulation . A great deal of success with methods of predicting 
through computer simulation of simple problems has led us to transfer 
simulation techniques to more complex problems. But although much useful 
work has been done in simulating small uncomplicated environments (such as 
factory operations) , there is little evidence to convince skeptics that this 
can be done on anything but the most macro scale for more complex, more dynamic 
social and technological societies. 

I am not saying we should not study the future by using simulation. 

I am only saying we should not delude ourselves into believing that we can 
get precise forecasts by applying rigorous techniques to poorly understood 
models of an environment. 



E.g., probability, the variables (exogenous and endogenous) that constitute 
an environment, trend extension, interactions among events, time, and 
tacit knowledge. 





Consider, for example, a societal mode], which includes a set of 
alternative policy decisions. One could try to simulate the making of 
policy decisions (1) in terms of mathematical formulas based on, say, a 
delphi analysis, or (2) by bringing together responsible policy-makers 
who would "play at" making policy in a simulated environment.* If the 
mathematical simulation could be shown to give (consistently) about the 
same results as the human one, it might be reasonable to say that that tool 
was of such a level of sophistication as to be a useful decision tool. If 
it was not able to foretell how people reacted when faced with a "real" 
situation, (i.e., if (3) were not able to consistently simulate (2)), we 
should conclude that the method should never be allowed to dictate policy, 
though it might be used to shed light on possible policies. 

2. Probability . In section II. C. 3 I mentioned the problem of CIM 
dealing with subjective probability estimates in a very strict mathematical 
way. I will elaborate a little on that now. 



Morris Cohen nicely describes the distinction between subjective and 
objective probability: 



Modern theories of probability may be generally characterized 
as either subjective or objective, i«e., dealing either with 
the character of our beliefs or opinions, or with the character 
of the objective evidence for these beliefs or opinions. . . . 

The whole modern psychologic tendency puts the emphasis on the 
mental phase of the beliefs called probable, and this is reinforced 
by popular discourse, which has many expressions for degrees of 
probability, such as, "Highly probable," "very likely," "almost 
certain," "improbable," "not at all likely," and others. We say, 

"I am almost certain;" "I am quite sure;" "I am convinced;" "It 
seems to me;" etc. But the whole tendency of modern logic and 
exact science demands a definiteness in probable judgments which 
does not seem to be offered by any difference in the intensity of 
belief .** 



See Bertrand de Jouvenel, The Art of Conjecture , translated from the 
French by Nikita Lary (Bas .c Books, Inc., New York, 1967), pp. 288-290, 
for an interesting discussion of the complementary use of computer and 
human simulation. 

**Morri3 R. Cohen, A Preface to Logic , (Meridian Books, Inc., New York, 1960), 
pp. 114-115. 
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A key concept in futures methods seems to be that of subjective probability. 
Very many techniques seem to involve guessing about the likelihood of 
events which can occur, usually only once. I just want to point out here 
the need to recognize that this type of probability estimate must necessarily 
involve a vague understanding of such concepts as wishful thinking, incomplete 
knowledge of both the system under study and how the system would respond to 
different conditions (as opposed to how one thinks it would respond) and 
many other ’’soft’’ variables. Bertrand de Jouvenel describes the "’mores 1 that 
our minds conform to in fore-thinking," which give "rise to very strong feelings 
of subjective certitude.”* He says that "because of its hidden nature this 
psychological process can have no place in a field of activity that is to 
be systematic, disciplined, justifiable and discussible." 

While I am not ready to discount any use of what de Jouvenel terms 
"pregnant forecasts," I will insist that a useful decision tool must not only 
recognize these, it must take them into account. A temptation might be to 
design a rigorous model that does not need to recognize the "soft" variables, 
but this would be a foolish path to follow if it is the "soft" variables 
that the future in question is made of. 

Some very lively and fruitful discussions on subjective probability 
have been going on for a long time among scholars and practitioners in many 
disciplines. It is generally agreed that we can often organize our thoughts 
better through the use of this concept than through simple informal intuition.** 
However, students of subjective probability have not yet determined just how 
it fits into statistical theory. We may be able to apply relevant research 



Bertrand de Jouvenel, op. cit . , p. 127. 

L. J. Savage, Statistical Inference (Wiley, 1959), is a good introduction 
to the important issues in the study of subjective probability and its 
potential influence on statistics. 



that has gone on in fields only peripherally related to the study of the 
future, but this must be done rigorously if it is to form the basis for 
accurate methods. 

3. Two kinds of knowledge, and the comparability of judgments . 

We have to face the possibility that we may never be able to get a handle 
on some concepts. I question whether it is possible, for example, to 
build into any forecasting technique a means for taking into account 
tacit knowledge. By tacit knowledge I re r er to the kind of knowledge 
which we may use in understanding a process and in making judgments, but 
which we may never be able to specify. 

We make judgments on the basis of a set of mental models that we 
use to understand the world. We may be able to articulate exactly some 
of these models, while others may be only subconscious and hence untellable. 
I will first consider the problems of making judgments on the basis of 
"tellable' 1 models. 

When we make a statement about a future phenomenon we are. making 
it on the basis of our personal understanding of that phenomenon. It is 
unlikely that our perception of any given phenomenon is exactly like some- 
one else's perception, though they may be very similar.* This is especially 
true if the phenomenon is complex and the judgment subjective, as is often 
the case in futures research. 

If perceptions of an entity are different, then it may not be possible 
to compare judgments about the entity, for suppose two people with very 
different perceptions about a future event each make some judgment about 



* 

See Weaver, Delphi as a Method for Studying the Future: Testing Som e 

Underlying Assumptions , for a discussion of the roles of perception 
and judgment in delphi. 
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the event. Even if the two judgments are exactly the same statements, 
they are not the same judgments . ; ’ £ 

Suppose, for example, that I am interested in when, if ever, 50% 
of elementary school learning will be done by television at home, and I 
am also interested in the most important consequence of this to society. 

And suppose Alpha, who (subjectively) sees schools as increasingly 
oppressive and hence as growing centers of unrest, thinks that policy-makers 
will look to home TV as an alternative to bringing children together 
in school all day. He also sees television as more of a mesmerizing than 
stimulating medium, so he expects children to be less excited and hence 
less concerned about whether the educational system is giving them what 
it ought to. Alpha's judgment: "50% of learning will be done by home TV 

by 1990 and as a consequence there will be fewer dropouts and greater 
satisfaction with the educational process. " 

Beta, however, sees greater movement toward open learning systems 
as schools become more flexible. Policy-makers will decide to offer 
optional elementary educational programs through home TV. Since certain 
subjects can be learned better by TV, Beta feels, many children and parents 
will opt for the TV programs. Beta's judgment: "50% of learning will be 

done by home IV by 1990 and as a consequence there will be fewer dropouts 
and greater satisfaction with the educational process." 

The judgments of Alpha and Beta are represented by exactly the same 
statements, but they are essentially different judgments since they are 
based upon completely different perceptions of the future. We should 
never say "Alpha and Beta agree on this future event." We should not 
even say that Alpha and Beta agree on the likely date when the event 



The distinction here is between the name of a thing (judgment) and the 
thing itself. I may call my cat Rosebud and you may call your dog 
Rosebud, but although they have the same name they are very diffei nt 
things, and neither of them is a rosebud. 
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will occur, for even though they use the same name for the date of 
occurrence, 1990, their estimates are inferred from contradictory premises 
and hence might even be contradictory. 

Ky point in this example is that if two people do have different 
perceptions, different mental models about the future there is no logical 
reason why their judgments should be comparable, even if their statements 
make them appear comparable. 



The problem of incomparable judgments is, in theory, soluble if we 
can only get the judges to make known the mental models upon which they 
base their judgments.* If this can be done, and this is one of the things 
the delphi method tries to do in a very superficial way**, then at least two 
advantages accrue. First, there results a better understanding of the 
different possible perceptions about the future; second, judgments which 
might otherwise seem to differ only in degree might be found to differ 
in kind. What does the latter imply for methods, such as delphi, which 
express a distribution of opinions in terms of a central tendency? How do 
you express a central tendency among incomparable estimates? What is the 
average of three cats, two dogs and a rosebud? 

Up to this point I have only discussed judgments based upon mental 
models that can be explained by the judger. These carry with them many 
problems, but even more perplexing is how to deal with those models which 



& 

There is an interesting discussion of how one might go about doing this 
in Stuart Sandow, The ¥ « ia gogical Structure of Methods for Thinking 
About the Future , especially his chapters on futures history analysis. 

Howard Johnson has noted that this superficiality is due not so much to 
the delphi concept as to "insufficient commitment on the part of the 
designers, referee(s), and panel members. Designing and executing a 
Delphi properly requires a great amount of effort, and in most cases 
the parties concerned are either unwilling or unable to give the required 
time and effort for unambiguous results." (personal communication). 
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I described above as untellable.* If certain of the perceptions upon 
which the judgments of Alpha and Beta are made cannot be made known, then 
it is impossible to know whether the judgments are ever comparable. 
Furthermore, since the constructs are subconscious we may never even 
know whether tacit perceptions exist (see last footnote), so strictly 
speaking we may never know whether judgments which might involve untellable 
perceptions are comparable. 

The foregoing discussion may seem irrelevant to real life situations, 
but I don’t think it is. I see too many arguments which are eventually 
resolved when the antagonists finally realize that they are not disagreeing 
on information or logic, but on their perceptions of a problem. And I 
see too many problems solved by "intuition" to believe that we do not hold 
valid logical mental models which we are not able to articulate. 

By examining these three problem areas, and similar ones, I think 
students of futures methodology can improve understanding and credibility 
of futures research methods. The future of futures research .may lie not 
in the development and dissemination of more of virtually the same techniques 
but in the identification of those basic concepts which keep cropping up, 
and in doing the research required to gain a defensible command of those 
concepts . 

Many of us feel intuitively that we can find ways to get a better handle 
on the future. If our intuition is wrong, future history may see us as 
another of those determined movements which tried to achieve the impossible. 
Leonardo da Vinci compared those who would build the perpetual motion machine 
with the alchemists who would change the base metals to gold**: "for 

[their oversight of] a little detail, [friction], everything was lost. . .1 



* 

See Michael Polanyi, Personal Knowledge , and The Tacit Dimension , in 
which he discusses the "inarticulate manifestations of intelligence by 
which we know things in a purely personal way." He even argues that there 
can be no knowledge without tacit knowledge. (p . 64) 

» ^ ^ 

Lasidae Reti, "Leonardo on Bearings and Gears," Scientific American , 
February, 1971. p. 103. 



remember that many people, from different countries, went to Venice with 
great expectation of gain to make mills in dead [still] water, and after 
much expense and effort, unable to set the machine in motion they were 
obliged to escape. 

"0 speculators about perpetual motion how many vain chimeras have 
you created in the like quest? Go and take ‘your place with, the seekers of 
gold!" 
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APPENDIX A 



Summary and Conclusions on Work Done at IFF, to April, 1970: 

A good summary of the work done at IFF is the paper The Use of 
Cross-Impact Matrices for Forecasting and Planning , by Rochberg 5 et. al . 
On pages 42-43 the major strengths of CIM are summarized: 



1. It prompts meaningful questions about interrelationships among 
events under test . 

2. It serves pedagogical purposes in raising these questions, in 
acquainting the expert participants with certain potential 
future occurrences, and in producing forecasts. 

3. It provides a new tool for determining the comparative plausibility 
of scenarios. 

4. It serves as a method of organizing judgment to provide a predictive 
device in areas in which exact causal relationships are extremely 
difficult to discern. 

5. It provides a method of stimulating certain policy actions (by 
comparing a "base-line" run with one in which the initial 
conditions are changed in order to simulate alternative action 
programs) . 

While I agree with (1) and (2), I hope this paper will indicate a need 
to question the validity of (3), (4) and (5). 



The IFF pape* indicates that research is continuing in the following 
areas : 



1. Development of analytic methods which would allow direct 
computation of the results (to replace the current Monte 
Carlo analysis) . 

2. Exploration of schemata under which probability adjustments are 
made in the negative sense for those events in the matrix which are 
judged not to occur. 

3. Introduction of sequential computations by which the matrix 
solution is accomplished in discrete steps, each building on the 
one earlier. 
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4. Review of historical data to discover a rational basis, if one 
exists, for forecasting diffusion time. 

5. Development of improved methods of using expert knowledge to fill 
the cells of the matrix. 



Of these problems, only (1) and (2) are directly addressed in this 
paper, but I feel that (3), (4) and (5) are equally important. I see 
this list as only a small subset of the questions that need to be addressed. 
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APPENDIX B 



The following is a reprint of most of Chapter I of IFF Report R-10, 
The Use of Cross-Impact Matrices for Forecasting and Planning , by 
Rochberg, Gordon and Helmer. ~ ~~ 

I . INTRODUCTION 



DEFINITION OF CROSS-IMPACT ANALYSIS 



A cross-impact matrix is an array consisting of a list of potential 
future developments and two kinds of data concerning these developments: 
first, the estimated probabilities that these developments will occur 
within some specified period in the future, and, second, estimates of 
the effect that the occurrence of any one of these events could be ex- 
pected to have on the likelihood of occurrence of each of the others . 

In general , the data for such a matrix are obtained by collating expert 
opinions derived through the use of methods such as the Delphi technique 
[1,2].* Such a matrix is analyzed in order to: 

- Revise the estimated probabilities of occurrence of each 
development in light of the expected cross impacts of 
other events on the list. 

- Discover how a change in the probability of occurrence of 
one or more events (by virtue of a technological break- 
through, a social change, a policy decision) might be ex- 
pected to change the probabilities of occurrence of other 
events on the list. 

Although the computational techniques used in a cross-impact analy- 
sis are sometimes complicated, the basic concepts underlying the tech- 
nique are straightforward. Suppose that the probability of occurrence 
of each of a set of developments by some year in the future has been 

forecasted. If these developments are designated D^, D D^ with 

associated probabilities P^, P P^ , then the question can be 

posed: "If P^ = 1 (that is, if D^ happens) , how do she other P_^ change?" 
In other words, we speak of a cross-impact effect if the probability 
that one event occurs varies either positively or negatively with the 



* 

Numbers in square brackets refer to publications cited in the 
list of references presented at the end of this Report. 
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occurrence or non-occurrence of other events. Assume, for example, that 
the following developments, with associated probabilities, were forecasted 
for a given year : 

Development Probability 

1. One-month reliable weather .4 

forecasts 

2. Feasibility of limited weather .2 

control 

3. General biochemical immuniza- .5 

tion 

4. Elimination of crop damage .5 

from adverse weather 

These events might then be arranged in matrix form (as in Figure 1) . 



Then the probability of 



If this development were to occur: 
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Figure 1 - BASIC FORM OF A CROSS- IMPACT MATRIX 
The upward arrows indicate an increase in probability. (Decreases would 
have been indicated by downward arrows.) Thus, if D^, "Feasibility of 
limited weather control", were to occur.- D , "One-month reliable weather 
forecasts", would become more probable, as noted by the upward a: row. 
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This kind of array is called a "cross-impact matrix". 

Interactions between events are much more complex, of course, than 
those tha. can be indicated by an arrow. The arrow denotes only that 
there is a linkage between events and the direction of the influence one 
event has on another. In addition, it is necessary to identify the link- 
age strength (how strongly the occurrence or non-occurrence of on ' event 
influences the probability of another) and the diffusion time (how long 
an interval is required after the occurrence or non-occurrence of one 
event before another event is influenced) . 

Once the arrows have been replaced by the requisite numerical data 
and the relevant formulas have been developed for calculating the changes 
in probabilities, such a matrix can be analyzed on a computer. Details 
of the techniques of analysis are presented later. Briefly, the follow- 
ing steps are involved: 

1. Assessing the potential interactions (that is, the cross impacts) 
among individual events in a set of forecasts in terms of : 

- Direction, or mode, of the interaction. 

- Strength of the interaction. 

- Time delay of the effect of one event on another. 

2. Selecting an event at random and "deciding" its occurrence or 
non-occurrence on the basis of its assigned probability. 

3. Adjusting the probability of the remaining events according to 
the interactions assessed in step 1. 

4. Selecting another event from among those remaining and deciding 
it (using its new probability) as before. 

5. Continuing this process until all events in the set have been 
decided. 

6. "Playing" the matrix in this way man/ times jo that the proba- 
bilities can be computed on the basis of the percentage of times 
that an event occurs during these repeated plays. 

7. Changing the initial probability of one or more events and re- 
peating steps 2 to 6. 

By comparing the initial probabilities and those generated in step 
6, it is possible to determine how the initial prob _cies light be 
modified to take into account the cross impacts of other events on the 
list. By comparing the results of jle.p <r> and step 7, it is possible to 
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determine how a change in the probability of occurrence of one or more 
events would, through the cross impacts, affect the probability of the 
other events. 



QUANTITATIVE TECHNIQUES 

To ascertain the cross impacts, the quantitative nature of the in- 
teractions must be specified. The following brief discussion presents 
the formulas that are used and the reasons why these formulas have been 
chosen. (An alternative computational method is presented in Appendix A.) 

Suppose that is the probability of D_^ before the occurrence of 
and that P_^ ! is the probability of D_^ some time after the occurrence of 

D . Then : 
m 

P. ' = f (P . , M, S, t , t) , (1) 

11 m 

where : 

M is the connection mode, 

S is a measure of the strength of connection, 

t is the time of the occurrence of D , and 
m m 

t is the time in the future for which the probabilities 

are being estimated. 

We know that both P. and P.' must lie between 0 and 1; furthermore, 

i i 

where the occurrence of an event increases or decreases the likelihood 

of occurrence of another event, if P_^ = 0, P^' must equal 0, and if 

P. = 1, P. ' must equal 1 (as indicated in Figure 2) . 
ii 





Hgure 2 - GRAPH OF A POSSIBLE CHOICE FOR 
THE FUNCTION IN EQUATION 1 
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When t - t , no time is allowed for the adjustment of probability of 
m 

P. to P , 1 , so P, must equal P.' (as indicated in Figure 3). 
iir i 




Figure 3 ~ THE FORM OF THE FUNCTION IN 

EQUATION 1 OF t = t 

m 



Above the diagonal F 1 > P. and below P. 1 < P, ; thus, in the area above 

i i i i 

the diagonal the impact of an event will be to enhance the probability 
of occurrence of another; below, it will be to inhibit this occurrence. 

It is reasonable to assume that the relationship between P^ and P^' 
is monotonic. As a first approximation we will assume that the relation 
is quadratics 

P, ' = a P, 2 + b P, + c. (2) 

i i i 

Then, substituting known end conditions, we obtains 

P, ' = a P . 2 + (1 - a) P, , (3) 

ii i 

or equivalently : 

P. ' = P. - aP . (1 - P. ) . (3') 

i i i i 

For the inhibiting case s 

0 < a < 1. (4) 



and for the 



enhancing case : 



(5) 
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-1 < a < o. 



The question still remains as to how t , t, and S are related to a, 

in 

Although greater sophistication is possible/ we make the simple assump 
tion that the relation is linear: 

? 

t - t 

a = kS - - - — / (6) 

where : 

k is +1 or -1 as determined by the mode/ 

S is a number between 0 and 1 , a smaller number representing 
weaker strength (zero designating an unconnected pair) , and 

t's are as previously defined. 

Mow substituting back into Equation (3) : 

t - t 

P,' - p. - P(1 - P) ks --- — — . (7) 

ir t 



FIRST APPLICATIONS 

The cross-impact method was developed by T. J, Gordon and 0, Helmer; 
it was first tested experimentally by Gordon and Hayward [3] , Gordon and 
Hayward considered two cases in detail. One was historical/ involving 28 
events judged relevant to the decision to deploy the Minuteman missile 
system; the other was futures-oriented/ involving 71 events forecasted to 
occur within the next 20 years and considered likely to have a bearing on 
future transportation services. In both cases, the events were arrayed 
in a matrix and estimates were provided for the direction/ strength, and 
time-phasing of the effects of the events on each other. Each matrix was 
then run 1000 times on a computer, according to the steps outlined earlier, 
and the results were averaged to produce new estimates of the probability 
of occurrence of the events in the matrices. The probability shifts iden- 
tified in this way provided a measure of the combined cross-impact effects 
implicit in the original matrix. 

The situation depicted by the results in the Minuteman analysis in- 
dicated that the mutual interaction of the events strongly enhanced the 
likelihood of a decision to deploy the system. Additionally, a ranking 



of the events in terms of their final, cross-impacted probabilities pro- 
vided the ingredients of a scenario quite descriptive of the technolog- 
ical and political environment of the 1950s, Thus, despite the simplifi- 
cations made in this case, the findings were consistent with what actu- 
ally occurred. 

The probability shifts apparent after the transportation matrix had 
been run pointed to substantive issues of considerable interest, since 
there were several significant changes in the original values, but the 
primary findings were methodological. Particular attention was given to 
testing the sensitivity of the cross-impact effects to changes in the 
initial probability levels, thus simulating the influence of conscious 
policy decisions or of unexpected breakthroughs. The initial probability 
assignment of certain events was raised arbitrarily by 20 percent, but in 
no case allowed to exceed ,95, When the new run was compared with the 
earlier "norm", there were a number of differences. Most of them were 
intuitively plausible, although .n at least one case a change occurred 
that could not readily be explained, thus suggesting that cross-impact 
analysis may also have a value in indicating unexpected causal linkages 
among particular events, •• 



REFERENCES 



1, N, C, Dalkey and O, Helmer, "An nocperimental Application of the 
Delphi Method to the Use of Experts," Management Science, Vol. 9, 
No, 3 (April 1963) . 

2, Selwyn Enzer, A Case Study Using Forecasting As a Decisionmaking 
Aid, IFF Working Paper WP-2, The Institute for the Future, December 
1969. 

3, T, J. Gordon and H, Hayward, "Initital Esqperiments with the Cross- 
Impact Matrix Method of Forecasting," Futures, Vol . 1, No. 2 
(December 1968) . 




B- 

57 






M6Q6IU 



APPENDIX C 



CPGj « 1 
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INPUT: t = NUMBER OF EVENTS 

g - NUMBER OF CPG TO BE 
GENERATED 



Algorithm referred to in section B.2 for generating CPG f s 



ml means the permutation vector 1, 2, 3 
def ined , the pr oduc t I * 2 * 3 * * * e , 



, , ,e, rather than, as usua 
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